-
Notifications
You must be signed in to change notification settings - Fork 30
INTPYTHON-527 Add Queryable Encryption support #329
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
django_mongodb_backend/management/commands/get_encrypted_fields_map.py
Outdated
Show resolved
Hide resolved
django_mongodb_backend/management/commands/get_encrypted_fields_map.py
Outdated
Show resolved
Hide resolved
django_mongodb_backend/management/commands/get_encrypted_fields_map.py
Outdated
Show resolved
Hide resolved
django_mongodb_backend/management/commands/get_encrypted_fields_map.py
Outdated
Show resolved
Hide resolved
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
ea5bb5c to
7404286
Compare
043a445 to
f9cbb3d
Compare
| key_vault_collection = client[key_vault_db][key_vault_collection] | ||
| # Create partial unique index on keyAltNames. | ||
| # TODO: find a better place for this. It only needs to run once for an | ||
| # application's lifetime. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe in the app's migrations ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I considered it (and it's still an option), but I think it's not ideal for the user to have to manage the operation.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It is not ideal and so I was hoping there was a way to put it in a base class somewhere.
a697cae to
a2d86ad
Compare
9748263 to
9f9507d
Compare
django_mongodb_backend/management/commands/showencryptedfieldsmap.py
Outdated
Show resolved
Hide resolved
| - name: Verify MongoDB installation | ||
| run: | | ||
| mongosh --eval 'db.runCommand({ connectionStatus: 1 })' | ||
| - name: Verify mongocryptd is running | ||
| run: | | ||
| pgrep mongocryptd |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What was your thinking here? Were these steps copied from another project? Wouldn't the start server / mongocryptd commands fail with a suitable exit code if they didn't start?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
POC developed here: https://github.com/aclark4life/test-supercharge-action. The verify steps can probably come out.
| - name: Start local Atlas | ||
| working-directory: . | ||
| run: bash .github/workflows/start_local_atlas.sh mongodb/mongodb-atlas-local:7 | ||
| run: bash .github/workflows/start_local_atlas.sh mongodb/mongodb-atlas-local:8.0.15 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I guess we should have a separate job for Atlas 8 so we still test with Atlas 7, although it's useful to keep it like this for now so we can see the modifications.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we care about 7 outside of QE? If not, let's just test 8. Still thinking about going the other direction and supporting query, queryPreview, etc. But in this one case I think it's reasonable to cling to non-preview and versions that support query.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes. In fact, based on the driver policy, we have to support MongoDB 6 until July 2028. I've argued that since Django 5.2 is supported until April 2028, we could make Django 5.2 the last version to support MongoDB 6. This is similar to Django's version support for its built in databases. In any case, it would be nice to finalize and document a MongoDB version support policy.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
OK in that case to remove as much ambiguity as possible, and to support as much MongoDB as we can, how about if we go back to 7 for QE and document rangePreview. I haven't thought about the MongoDB support policy in the context of this project, but what you propose sounds fine. Maybe open a PR to the docs with that policy defined so we can discuss there and merge.
| mongosh --version | ||
| - name: Install mongocryptd from Enterprise tarball | ||
| run: | | ||
| curl -sSL -o mongodb-enterprise.tgz "https://downloads.mongodb.com/linux/mongodb-linux-x86_64-enterprise-ubuntu2204-8.0.15.tgz" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we use a variable for 8.0.15 so it's not hardcoded in so many places? (I have to research to answer.) Same for ubuntu2204, probably. This action uses ubuntu-latest which is 24.04 I believe.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| Queryable Encryption can be used with MongoDB replica sets or sharded | ||
| clusters running version 8.0 or later. Standalone instances are not |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Users may wonder why it says 8.0 when the MongoDB documentation says 7.0. Even if we only test with 8 because of the range/rangePreview issue, I think our implementation should work on 7.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
True, but I'm still inclined to steer away from Preview and only declare support when a query type makes it out of preview.
| "OPTIONS": { | ||
| "auto_encryption_opts": AutoEncryptionOpts( | ||
| key_vault_namespace="encryption.__keyVault", | ||
| kms_providers={"local": {"key": os.urandom(96)}}, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Using os.urandom(96) generates a different key each time Python starts which is problematic in a local project, except when running tests. Need to explain this or use a different approach.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There is an admonition about it here: https://django-mongodb-backend--329.org.readthedocs.build/en/329/howto/queryable-encryption/#configuring-the-databases-setting
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
My point here (as we've discussed) is that it's not appropriate to use a random key that changes each time the web server restarts, a management command, etc. Should we hardcode a particular bytestring that os.urandom(96) generates?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
b0f79ef to
f275f0a
Compare
f275f0a to
71a611d
Compare
c0b88bb to
a6d0bef
Compare
Previous attempts and additional context here:
INTPYTHON-527 Add Queryable Encryption config #318
INTPYTHON-527 Add queryable encryption support #319
INTPYTHON-527 Add Queryable Encryption support #323
Add test for "Encrypted fields found" error (ensure this exception still happens)
Add check for model schema not matching encrypted fields
Document key_vault_namespace must be encrypted db
Document that fields within EmbeddedModelArrayField can't be encrypted
Document workflow: